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DETAILED ACTION 
Response to Arguments 

1 . Applicant's arguments, see Pages 1-5, filed 24 March 2005, with respect to all 
matters have been fully considered and are persuasive. The objections to the drawings 
and specification have been withdrawn. 

2. The rejections to claims 1-26 under 35 U.S.C. 101 have been withdrawn after 
further consideration of applicant's arguments (see pages 2-3). 

3. Applicant's arguments, see Page 3, with respect to the rejection of claim 27 
under 35 U.S.C. 101 has been fully considered but is not persuasive. Under MPEP 
2106, a computer program must be tangibly embodied on computer-readable media in 
order to be statutory. Claim 27 as it stands currently amended still merely recites a 
computer program per se, which does not fall under the relevant safe harbor provisions, 
either the older ones established by Abele and AT&T or the newer one established by 
State Street. In order to overcome this rejection, applicant must amend preamble of 
the claim to recite that the computer program is tangibly embodied on computer- 
readable media. 

4. Applicant's arguments - see Pages 2-3 specifically -- with respect to the 
rejections of various claims under 35 U.S.C. 112, first and second paragraphs, have 
been fully considered and are persuasive. All rejections under 35 U.S.C. 112 stand 
withdrawn due to applicant's amendment. 

5. Applicant's arguments, see Pages 3-6 with respect to the rejections of the 
various remaining claims, namely 1, 3-5, 8, 12-13, and 15-27 under 35 U.S.C. 102(b), 
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102(e), and 103(a) variously all stand withdrawn in view of applicant's amendments to 
the claims. 

However, upon further consideration in light of the amendments, new grounds of 
rejection follow below. 

Claim Rejections - 35 USC § 101 

6. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

7. Claim 27 is rejected under 35 U.S.C. 101 because it is directed to non-statutory 
subject matter, e.g. a computer program per se. This rejection was explained above in 
the Response to Arguments section. 

Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

9. Claims 1 , 3, and 8 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Thomas (Thomas et al. "ARQuake: An Indoor/Outdoor Augmented Reality First 
Person Application." (2000))('T2000') in view of Piekarski (Piekarski et al. "Integrating 
virtual and augmented realities in an outdoor application") and Miyashita et al (US 
6,057,856). 

10. As to claim 1, 
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An information presentation apparatus comprising: 
-User operation input unit, adapted to input an operation of a user (T2000 - "two button 
input device" (section 3, pg. 141))(Piekarski Fig. 3, the Phoenix forearm 
keyboard)(Miyashita computer mouse and keyboard, see Fig. 5 where such elements 
are listed as 29a and 29b as well as microphone 26); 

-User viewpoint position and orientation measurement unit, adapted to measure a 
position and orientation at a user's viewpoint (T2000 section 2.3, where the specific 
tracking mechanisms are disclosed (and a head orientation measurement unit); further 
the fiducial markers (T2000, Fig. 3, pg. 143) serve to correct registration areas in and 
close to buildings)(similar mechanisms in Piekarski)(Miyashita 2:10-55, specifically 
3:25-45, where the virtual world keeps track of user avatar location); 
-Input unit, adapted to input viewpoint position and orientation information of an other 
user (Miyashita Figs. 29-31 where the radar screen is shown, Figs. 9-13 where the user 
can view the viewpoint of the avatar of the other user, Piekarski Figs. 6-8 where each 
wireless base unit shown in Figure 4 would know its own location and would obviously 
be sharing that information with the system and such system would show that location 
when combined with Miyashita because the system shows the location of known objects 
in the simulator (e.g. CHOPPER0) and it would be obvious to show the location of other 
users (also, monsters are shown in T2000)) 

-Model data storage unit, adapted to store virtual world model data (T2000 section 2.3 - 
the hard drive in the notebook computer (hard drive is inherent in notebook PC); Quake 
inherently stores virtual world data and data necessary to generate a virtual world); 
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-Annotation data storage unit, adapted to store annotation data (Piekarski section 2.3 - 
the hard drive in the notebook computer (hard drive is inherent in notebook PC); as 
shown in Figs. 6, 7, and 8b data from the virtual world is displayed on the HUD - that is 
the augmented view, which shows data annotated by the game, and annotated overlay 
information is shown. In Fig. 7, range information and markers are shown as 
well.)(Miyashita inherently has virtual world model data, as in 2:10-55 and Figs. 1-4); 
-Virtual image generation unit, adapted to generate a virtual world image by using 
measured results of the position and orientation of the user's viewpoint (Piekarski 
section 3.1 - the notebook PC (the graphics card would provide this functionality) - 
inherent in notebook computer. )(T2000 2.3 again - that is, the notebook computer 
would supply this functionality)(Miyashita Figs. 9-14, where the virtual world is 
generated from the point of view of the avatar as shown therein); 
-Annotation image generation unit, adapted to generate an annotation image from the 
annotation data, based on position and orientation information of the user and the 
viewpoint position and orientation information of the other user; (Piekarski section 2.3 - 
the hard drive in the notebook computer (hard drive is inherent in notebook PC); as 
shown in Figs. 6, 7, and 8b data from the virtual world is displayed on the HUD - that is 
the augmented view, which shows data annotated by the game, and annotated overlay 
information is shown. In Fig. 7, range information and markers are shown as 
well.)(Miyashita inherently has virtual world model data, as in 2:10-55 and Figs. 9-13, 
where in 2:10-55 and 3:20-45 as well as the abstract and 11:13-13:30; users can chat 
as in 27:60-28:5, where the user can talk to each other, and further in 10:8-25 it is 



Application/Control Number: 10/626,590 Page 6 

Art Unit: 2672 

disclosed that the user enters positional information and it is superimposed on the user 
or item (3:20-45)) 

-User viewpoint image input unit, adapted to capture an image of the real world viewed 
from the user's viewpoint (T2000, section 7, a camera for use with the system is 
disclosed. Also, given that the fiducials could be recognized, discussion in sections 4.2 
and 4.3, and fiducials shown in Fig. 3, a camera is inherent); 
-Composite unit, adapted to composite the image of the real world, the virtual world 
image, and the annotation image; and (T2000 heads-up display for user as disclosed in 
section 2.3, overlay view and a sample view is shown Fig. 2, for example, with virtual 
world information displayed )(Piekarski also inherently has a compositing unit for 
combining the virtual world and annotation with the real world so that the overlay will be 
accurate, for example in Figure 4 and Figure 5, and since the user is wearing a see-thru 
HUD it is required) 

-Image display unit, adapted to display a composite image acquired by said composite 
unit (T2000 heads-up display for user as disclosed in section 2.3, overlay view and a 
sample view is shown Fig. 2, for example, with virtual world information displayed). 

Reference T2000 teaches all the limitations of the claim except explicit use of 
annotation database and other details on annotation (but that is fairly obvious in the 
presence of the extra data from the game shown in the overlay in Fig. 2). Reference 
Piekarski discloses the use of annotation and overlay techniques clearly in Figs. 6 and 7 
that show the augmented reality view with annotated objects and their information. It 
would have been obvious to one having ordinary skill in the art at the time the invention 



Application/Control Number: 10/626,590 Page 7 

Art Unit: 2672 

was made to combine the augmented reality system with camera of T2000 with the 
annotated augmented reality system of Piekarski, since the non-camera system of 
Piekarski could achieve an accuracy of 5 meters in most cases, but the incorporation of 
the fiducials via the camera in the later version of the system improved accuracy to 
10cm indoors (and GPS is said to not function indoors - see T2000, section 4 for 
discussion). Further, the Piekarski reference is by the same research group at the 
same university in Australia, utilizing the same (Tinmith platform) computer platform, 
and share common authors with the T2000 reference. 

Reference Miyashita clearly teaches a virtual world, where users can 
communicate with each other and positional information concerning other users and 
objects is overlaid upon them, and they can communicate with each other. Very clearly, 
this represents the same idea as taught in Piekarski, with multiple users all being aware 
of each other and being able to communicate with each other and see each other's 
positions. The combination of Miyashita would allow the users to know more accurately 
where each other is and communicate via chat and other methods, and see each 
other's entries and information in the form of annotation as in Piekarski. Further, 
Miyashita clearly allows the users to see where each other are and furthermore to see 
the viewpoint of other users such that annotation generated by other users and their 
position are shown to each other (see Figs. 9-13 for example, and as cited above in 
1 1 :1 3-1 3:30). Motivation for combination with Miyashita would be so that users could 
communicate more effectively, have positional and annotation information 
superimposed on them and around them, and allow users to comment and chat as well 
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as annotate objects. Fundamentally, it would be obvious to modify the system of 
Piekarski such that users could interact and provide chat and annotations to objects to 
other individual - such is extremely well known in the VR art (see for example Jung et 
al, "Annotating and Sketching on 3D Web Models", where the users can put virtual post- 
it notes and comments on structures and many other features (pages 95-97)). 
11. As to claim 3, 

An information presentation apparatus according to claim 1 , wherein said input unit 
inputs, in addition to the viewpoint position and orientation information of the other user, 
identification information of the other user and operation information of the other user. 

Piekarski teaches various information in Figs. 6-8, such as in Fig. 8(b) where the 
location of choppers are shown overlaid in the HUD view, along with their identification 
and position information, which would represent other users or objects. ARQuake does 
not teach this specific limitation. Miyashita clearly teaches in for example Fig. 29 where 
the user location screen is shown overlaid in the upper right hand corner and each user 
has identifying information (e.g. the letters 'A' and 'B' overlaid on them to identify them) 
and such information concerning each user is available, e.g. name, location (IP 
address), etc. (see 19:30-20:50 for example, and 31 :55-32:50, where it is taught that 
information concerning other objects can be overlaid as well, and users can chat and 
communicate with each other. Again, also, in Figs. 9-13 and as explained above, each 
user can selectively choose to view the viewpoint of said another user so as to allow the 
user to see what object for example a customer is viewing to allow them to provide 
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more accurate help to the other user as recited in claim 1 above. Motivation and 
combination is taken from the parent claim and herein incorporated by reference. 
12. As to claim 8, 

An information presentation apparatus according to claim 1 , wherein the annotation 
data includes position and pose information of an object, discrimination information of 
the object, and text, symbol and image information for indicating information of the 
object. 

Reference T2000 does not specifically teach annotation data in this format (e.g. 
Quake will display player health, armor, etc., overlaid on the game screen (see Fig. 2 
T2000). Reference Piekarski clearly shows in Figs. 6, 7, and 8B the use of annotated 
data. Further, the system very clearly (see section 2.2) allows users to add entities and 
relative position information, as well as information on the "nature" of the entity being 
added. Piekarski clearly shows the position, speed, etc., of other entities in the 
environment in the overlay view (see Figs. 6, 7, and 8B), and as discussed above, 
would obviously convey that information (and logically any other required to convey 
pose / position information) about other users in the system, and this would include 
discrimination information. As shown in Figure 7, the objects have names and 
numbers; obviously, these could prima facie be separated so as to provide text. The 
symbol provided could be the arrows of Figs. 6, 7, or 8B, and the image would be, for 
example, the avatar shown in T2000 (Fig. 2) that shows the player. It would have been 
obvious to one having ordinary skill in the art at the time the invention was made to 
combine the augmented reality of T2000 with the annotation of Piekarski, as this would 
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allow the user to obtain information on the nature of entities (e.g. enemy units) and 
transmit the nature of new ones spotted to other soldiers / collaborators, etc. (see 2.2 
and 6 in Piekarski). Also, Miyashita clearly includes information on objects and overlays 
that on the map as well (see for example 32:20-50). Motivation and combination is 
taken from the parent claim and herein incorporated by reference. 

13. Claims 4 and 5 are rejected under 35 U.S.C. 103(a) as unpatentable over T2000 
in view of Piekarski and Miyashita as applied to claim 1 above, further in view of 
Piekarski (Piekarski et al. "Tinmith-Metro: New Outdoor Techniques for Creating City 
Models with an Augmented Reality Wearable Computer. ")('P2001 '). 

14. As to claim 4, 

An information presentation apparatus according to claim 1 , wherein the virtual world 
model data includes three-dimensional coordinates of vertices of a polygon of a virtual 
computer graphics (CG) object arranged in the virtual world, structure information of 
faces of the polygon, discrimination information of the CG object, color information, 
texture information, a size of the CG object, and position and pose information 
indicating the arrangement of the CG object on the virtual world. 

References T2000 and Piekarski do not explicitly teach all the limitations of the 
claims. Reference Piekarski does teach the overlay of a CG object with position and 
pose information (e.g. the helicopter shown in Fig. 8C would show up on the user's 
display, and be overlaid with the information shown in Fig. 7 / 8B that shows distance 
and name of the object - it would be an obvious variation to show all data at once), but 
does not teach all the information about structures of the polygons, etc. 
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Reference P2001 - a more advanced version of the Tinmith platform used in 
T2000 and Piekarski - teaches the use of the system in designing building for urban 
environments, with the accordant information about polygons etc. As shown in P2001 
Fig. 2A-2C, different versions of the same building, made of polygons, are shown. Each 
polygon is constructed from lines and planes (Fig. 2A) and then the details of each 
polygon (face, etc.) are shown in Fig. 2B (in the left margin of the image, specific 
information on the faces is shown). Fig. 2C shows overall orientation of the building 
with more specific attributes shown. As shown in P2001 Fig. 2B, each face is shown 
with the texture and color applied, with that information also supplied in the left margin, 
since the faces are being constructed and illustrated during the design process. P2001 
teaches in section 3 different operations that are used to construct buildings and objects 
in 2-D and then extend them to 3-D or simply form them in 3-D. On section 7, pg. 37, 
P2001 teaches that the Tinmith platform can render the scenes into primitive triangle 
lists or VRML files. As one of ordinary skill in the art is doubtless aware, Virtual Reality 
Markup Files contain all the vertices of objects described therein so that the objects can 
be accurately rendered. Given that, and Fig. 2B, it would be obvious to show the 
specific coordinates of the vertices as argued by applicant. Fig 2C represents final 
coordinate information of the object in the real world - also Fig. 3, which coupled with 
the coordinate information shown in Fig. 2B of the faces and the position and pose 
information revealed in Piekarski, would make it prima facie obvious to have the pose 
and position information of the structure shown, as well as any additional information 
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required. The specific information in Figs. 2A and 2B during the construction process 
show size of the various faces of the polygons / buildings, etc. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the construction systems of P2001 , since system of P2001 allows users to create 
objects for use in the augmented reality worlds of T2000 and Piekarski (common 
technology platform)(see sections 3 and 4 for specific construction methods). 
15. As to claim 5, 

An information presentation apparatus according to claim 1 , wherein 
-Said model data storage unit stores real world model data, and (T2000 section 2.3 - 
the hard drive in the notebook computer (hard drive is inherent in notebook PC). Prima 
facie all the systems - T2000 and Piekarski - store real world model data, particularly 
for ARQuake, since they have to know the location of the user in the physical space so 
as to known where to register monsters, etc. Quake inherently stores virtual world data 
and data necessary to generate a virtual world) 

-The real world model data includes three-dimensional coordinates of vertices of a 
polygon of an object existing in the real world merged with the virtual world, structure 
information of faces of the polygon, discrimination information of the object, a size of the 
object, and position and pose information indicating the arrangement of the object. 

See the rejection above for claim 4. The only additional limitation here is the 
limitation that such real world model data includes three-dimensional coordinates, etc. - 
the same limitations as for claim 4. They are all addressed above - and the augmented 
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reality system shown in Piekarski has such elements merged anyway, as the helicopter 
(Fig. 8A and 8C) is present in the system as overlaid. Particularly in Piekarski, since no 
camera is present, the real world model must prima facie include the virtual world data 
in order to generate the virtual camera views discussed therein (see section 2.2). Also, 
in T2000 the Quake monsters are displayed "in spatial context with the physical world" 
which indicates that the data does exist in the real world model as recited above in 
claim 5. Thusly, it would be obvious to use the augmented reality of P2001 with the real 
world model data, as the application is for modeling objects in the real world, and all 
three references utilize the same Tinmith technology platform. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the construction systems of P2001, since system of P2001 allows users to create 
objects for use in the augmented reality worlds of T2000, Piekarski, and Miyashita 
(common technology platform)(see sections 3 and 4 for specific construction methods). 

16. Claims 12, 15-18, 22-23, and 27 are rejected under 35 U.S.C. 103(a) as 
unpatentable over T2000 in view of Piekarski and Miyashita as applied to claim 1 
above, and further in view of Thomas. 

17. As to claim 12, 

An information presentation apparatus according to claim 1 1 , wherein the annotation is 
text, symbols, or images. 

Reference T2000 does not teach this limitation. Reference Piekarski teaches 
some of the limitations, while reference Thomas teaches the rest. See the above 
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rejection to claim 1 1 . Miyashita at least hints at this limitation with the overlaid 
characters on users in Figs. 29-30, and the note in 32:20-50 that other objects could 
have three-dimensional icons overlaid for them. 

Piekarski teaches in Figure 7 that the objects have names and numbers; 
obviously, these could prima facie be separated so as to provide text. The symbol 
provided could be the arrows of Figs. 6, 7, or 8B, and the image would be, for example, 
the avatar shown in T2000 (Fig. 2) that shows the player. Reference Piekarski teaches 
the use of multiple users and as well as simulated objects (planes, ships, tanks, etc.). 
Each object/unit, as shown in the overlays in Figs. 6, 7, and 8B has a label with a 
number (e.g. CHOPPER0, etc.) where this could obviously be split to be CHOPPER 1 , 
with the name and number separate, thus meeting that recited limitation. It would have 
been obvious to one having ordinary skill in the art at the time the invention was made 
to combine the augmented reality of T2000 and the annotation of Piekarski with the 
arrows of Thomas, as this would allow the user to obtain information on the nature of 
entities (e.g. enemy units) and transmit the nature of new ones spotted to other soldiers 
/ collaborators, etc. (see 2.2 and 6 in Piekarski) and other players (Thomas). 
18. As to claim 15, 

An information presentation apparatus according to claim 1 , wherein said 
annotation image generation generates, when a target object of the other user is 
outside a visual range of the user, an annotation indicating a direction of the target. 

References T2000 and Piekarski as well as Miyashita do not explicitly teach this 
limitation. Reference Piekarski shows in Fig. 8 the use of an arrow pointing at the entity 
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directly in the field of view of the user, and a waypoint is shown off to the left in Fig. 7. 
All the targets in Fig. 7 have a range and direction associated with them. Reference 
Thomas teaches the use in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow 
is taught to mimic the behavior of the player virtual avatar, e.g. it points in the direction 
the user is looking and is noticeable by the other players. Thomas suggests is that the 
user would be able to control visibility and location of the pointer. In section 2.2 Thomas 
disclose the use of a sophisticated head-tracking system for pose information. The 
arrow would prima facie follow the user's head orientation. Thomas teaches that it is 
visible to other users to guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that 
same page). 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target annotation 
in Fig. 7 and Thomas would allow other users / soldiers / players (in the case of T2000) 
to see the targets of interest and guide their attention to them when the targets were 
outside their field of view. 
19. As to claim 16, 

An information presentation apparatus according to claim 1 , wherein said annotation 
image generation unit generates an annotation for discriminating a target object of the 
other user. 
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See the rejection for claim 15, as this claim is a substantial duplicate - the only 
difference is that the words "outside a visual range" has been substituted omitted. It 
would prima facie obvious have the arrow of Thomas point at objects of interest to the 
user, as Thomas is silent on field-of-view issues, meaning that it would be obvious to 
have the arrow pointing at targets of interests to the user, including when the target was 
inside the field of view of another user, as Thomas teaches that the arrow / target 
indicator can be pointed at the specific target by the user. As noted in the rejection for 
claim 15, Piekarski teaches notation for targets shown on the overlay anyway. It would 
have been obvious to one having ordinary skill in the art at the time the invention was 
made to combine the combine the augmented reality of T2000 and the annotation of 
Piekarski with the arrows of Thomas, since Piekarski discloses tracking arrows to 
indicate objects (e.g. the helicopter in Figs. 8B and 8C) and provide target annotation 
data as shown in Fig. 7 and Thomas would allow other users / soldiers / players (in the 
case of T2000) to see the targets of interest and guide their attention to them when the 
targets were inside their field of view. 
20. As to claim 17, 

An information presentation apparatus according to claim 1 , wherein said annotation 
image generation unit generates an annotation for indicating the other user. 

References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
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in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). 

Further, it would be obvious to one of ordinary skill in the art that labeling multiple 
targets with the same color markers and annotations would be confusing and create 
visual clutter (it is well known in the human-computer interface art (HCI) that visual 
clutter makes it more difficult to perform depth processing, e.g. it makes the user 
inefficient.) Given this, it would be obvious to make the markers and annotations 
different colors for different targets so that they could be more easily picked out / 
discriminated when there are multiple targets or users visible to the user. This is 
standard procedure in video games. 

Further, reference Piekarski attaches annotations (e.g. CHOPPER0) to items in 
the HUD as shown in Figs. 6-8, especially Fig. 7 and Fig. 8b. Also, as in Figs. 29-31 in 
Miyashita, the reference teaches that avatars are labeled as 'A, 'B', et cetera. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and provide target 
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annotation data as shown in Fig. 7 and Thomas would allow other users / soldiers / 
players (in the case of T2000) to more easily determine their location and pick out their 
actions. 

21. As to claim 18, 

An information presentation apparatus according to claim 1 , wherein said annotation 
image generation unit has a function capable of controlling a generated annotation, by 
the user's operation input to said user operation input unit. 

The system in Piekarski clearly allows users to enter information using the 
forearm keyboard shown in Fig. 3, and transmits such data to other users and the 
system generally (as discussed in earlier rejections). Therefore, it would logical that the 
user could text-annotate the arrows, particularly if one user were based at a 
workstation, as Thomas discloses in section 5.1. Although voice communications are 
mentioned, data communications channels (e.g. the chat features of Quake (inherent to 
the software)) would be much easier to use for annotation purposes, etc. With a 
forearm keyboard, written annotation in addition to commands could be transmitted and 
seen by other users. 

Reference Thomas teaches the use in section 5 and in Fig. 1 1 of a 3-D pointing 
arrow. This arrow is taught to mimic the behavior of the player virtual avatar, e.g. it 
points in the direction the user is looking and is noticeable by the other players. 
Thomas suggests is that the user would be able to control visibility and location of the 
pointer. 
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Reference Miyashita further teaches that users can chat and interact (see the 
rejection to claim 1 , which is herein incorporated by reference, and also Figs. 29-31 
where it is shown that users can chat). 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the pointers of Thomas, since the use of controllable annotation would allow more 
effective conveyance of information on the nature of a target, and allowing users to 
control the annotation would allow them to communicate - visually, which would be very 
helpful in a multi-user game or simulation (as in Piekarski). 

22. As to claims 22 and 27 [claim 27 is a software implementation of method claim 
27, and the limitation of software is taught by T2000 and Thomas inherently in the use 
of the software on the computer, e.g. the Quake program, the use of the ARToolkit 
library, etc.], (The rejection to claim 1 is herein incorporated by reference in its entirety) 
An information processing method comprising the steps of: 

-Inputting viewpoint information of a user (T2000, section 7, a camera for use with the 
system is disclosed. Also, given that the fiducials could be recognized, discussion in 
sections 4.2 and 4.3, and fiducials shown in Fig. 3, a camera is inherent) (T2000 section 
2.3, where the specific tracking mechanisms are disclosed (and a head orientation 
measurement unit); further the fiducial markers (T2000, Fig. 3, pg. 143) serve to correct 
registration areas in and close to buildings)(similar mechanisms in Piekarski); 
-Generating a virtual world image according to the viewpoint information, by using 
previously held virtual world data (T2000 section 2.3 - the hard drive in the notebook 
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computer (hard drive is inherent in notebook PC); Quake inherently stores virtual world 
data and data necessary to generate a virtual world; Quake performs this limitation); 
-Inputting viewpoint information of an other user; (Miyashita Figs. 29-31 where the radar 
screen is shown, Figs. 9-1 3 where the user can view the viewpoint of the avatar of the 
other user, Piekarski Figs. 6-8 where each wireless base unit shown in Figure 4 would 
know its own location and would obviously be sharing that information with the system 
and such system would show that location when combined with Miyashita because the 
system shows the location of known objects in the simulator (e.g. CHOPPER0) and it 
would be obvious to show the location of other users (also, monsters are shown in 
T2000)) 

-Generating an annotation concerning an attention target based on the viewpoint 
information of the user and the viewpoint information of the other user (Thomas teaches 
the use in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic 
the behavior of the player virtual avatar, e.g. it points in the direction the user is looking 
and is noticeable by the other players. Thomas suggests that the user would be able to 
control visibility and location of the pointer.)(Clearly, Miyashita shows locations based 
on user locations as in Figs. 9-13 and 29-31); and 

-Generating an image obtained by synthesizing an image of a real world, generated 
virtual world image and the generated annotation. (Piekarski section 3.1 - the notebook 
PC (the graphics card would provide this functionality) - inherent in notebook 
computer.)(T2000 2.3 again - that is, the notebook computer would supply this 
functionality) (T2000 heads-up display for user as disclosed in section 2.3, overlay view 
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and a sample view is shown Fig. 2, for example, with virtual world information 
displayed). 

As discussed above, the various embodiments of the Tinmith platform presented 
in the above papers perform various aspects of this functionality. Piekarski clearly 
illustrates the annotation aspect and Thomas clearly establishes the attention target and 
arrow aspect. Very clearly, the user could indicate which target held their attention by 
shooting it with the gun or indicating it with the moving cursor of Thomas (both are 
common techniques in video game systems and well known in the art). See the 
rejection for claim 1 for the specific details of some of the systems, aside from the 
above noted reference locations for specific points in the claim. As discussed in the 
rejection to claim 18, the user could text-annotate or further mark the arrows to indicate 
attention targets. Further, the Quake program is known to construct the virtual world 
from one previously stored on disk (it is inherent). Finally, it is prima facie obvious that 
all the views are merged so that they are displayed in the HUD of the Tinmith system. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
and Miyashita with the pointers of Thomas, since the use of controllable annotation 
would allow more effective conveyance of information on the nature of a target, and 
allowing users to control the annotation would allow them to communicate - visually, 
which would be very helpful in a multi-user game or simulation (as in Piekarski). 
Further, all the other portions of the relevant explanation of how the viewpoint 
information of another user is incorporated, etc., is all covered in the rejection to claim 1 
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which has been incorporated by reference - the motivation and combination are also 
taken from therein for the combination of Piekarski, T2000, and Miyashita. 
23. As to claim 23, 

An information processing method according to claim 22, wherein said annotation 
generation step is adapted to generate, when a target object of the other user is outside 
a visual range of the user, an annotation indicating the direction of the target. 

References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). 
It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target annotation 
in Fig. 7 and Thomas would allow other users / soldiers / players (in the case of T2000) 
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to see the targets of interest and guide their attention to them when the targets were 
outside their field of view. 

24. Claim 1 3 is rejected under 35 U.S.C. 103(a) as unpatentable over T2000 in view 
of Piekarski and Miyashita as applied to claim 1 above, and further in view of Pham et al 
(US 6,259,396)('PhanV). 
As to claim 13, 

An information presentation apparatus according to claim 1 , wherein said annotation 
image generation unit recognizes a target object according to the operation of a user, 
and generates an annotation concerning the recognized target object. 

References T2000 and Piekarski do not explicitly teach the limitations of the 
claim. However, T2000 teaches the use of fiducials for distance and registration 
purposes for indoors and outdoors applications, which means that the camera and 
computer system are performing image processing and automatic target recognition 
(ATR) of the fiducial marks. Further, Piekarski teaches the use of battlefield simulators 
for military purposes, where automatic target recognition is a fundamental of the art 
(e.g. in signal processing for military applications, missile warhead guidance systems 
(e.g. uncooled and cooled IR focal plane arrays for heat-seeking missiles, etc).) In both 
Piekarski and T2000, pose / orientation tracking is present, which means that image 
processing is always being performed in the direction that the user is looking. 
Reference Pham teaches the use of an ATR algorithm that use a focus of attention 
(FOA) algorithm to highlight possible target locations in a full scene, e.g. perform ATR 
based on picking targets out as a human would (Pham 2:37-60). Miyashita at least 
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hints at this limitation with the overlaid characters on users in Figs. 29-30, and the note 
in 32:20-50 that other objects could have three-dimensional icons overlaid for them 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality and image processing systems 
of Piekarski, T2000, and Miyashita with the ATR FOA algorithms of Pham to allow 
discrimination of targets in the human field of view (including fiducials) in a "fast and 
efficient manner" (Pham 2:42). 

25. Claims 20-21 , 24, and 26 are rejected under 35 U.S.C. 1 03(a) as unpatentable 
over T2000 in view of Piekarski and Miyashita as applied to claim 1 [or other listed 
claims] above, and further in view of Tribes. 

26. As to claim 20, 

An information presentation apparatus according to claim 1 , wherein said annotation 
image generation unit generates, when the other user is outside a visual range of the 
user, an annotation indicating a direction of other user. 

The screenshots below illustrate the markers that the program generates to note 
players on the same and different teams. There are markers for players that are in the 
field of view but not visible if the location is known (e.g. teammates behind walls, behind 
hills, etc.)(The same concept as employed by the U.S. Army's Blue Force Tracker 
system, for example.) Further, people on the same team can designate targets that will 
be shown to each other using the targeting laser, and a little arrow will show up on 
either side of the screen indicating direction to the target. 
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References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). In Tribes, 
as shown in the screenshots above, each team has different color arrows to point to 
teammates and enemies, which convey information on player location. Reference 
Miyashita does not expressly teach this limitation. 
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It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas and Tribes, since Piekarski discloses 
tracking arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target 
annotation in Fig. 7 and Thomas would allow other users / soldiers / players (in the case 
of T2000) to see the targets of interest and guide their attention to them while indicating 
known friendly and unfriendly targets (Tribes) when the targets were outside their field 
of view. 

27. As to claim 21, 

An information presentation apparatus according to claim 1 , wherein said 
annotation image generation unit generates an annotation for discriminating the other 
user. 

This claim is a substantial duplicate of claim 20; therefore, the rejection for claim 
20 is incorporated herein by reference. The main difference is that "Tribes!" shows on 
the screenshots above that there are arrows pointing to friendly and unfriendly targets in 
the user's field of view (that is, other users are the targets). It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to combine the 
combine the augmented reality of T2000 and the annotation of Piekarski with the arrows 
of Thomas and Tribes, since Piekarski discloses tracking arrows to indicate objects (e.g. 
the helicopter in Figs. 8B and 8C) and target annotation in Fig. 7 and Thomas would 
allow other users / soldiers / players (in the case of T2000) to see the targets of interest 
and guide their attention to them while indicating known friendly and unfriendly targets 
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(Tribes) when the targets were inside their field of view. Further, Miyashita as in Figs. 
29-31 clearly indicates the position of other users with the radar image in the upper right 
hand corner, and as previously discussed in the rejection to claim 1 . 
28. As to claim 24, 

An information processing method according to claim 22, wherein said annotation 
generation step is adapted to generate an annotation for discriminating a target object 
of the other user. 

The screenshots below illustrate the markers that the program generates to note 
players on the same and different teams. There are markers for players that are in the 
field of view but not visible if the location is known (e.g. teammates behind walls, behind 
hills, etc.)(The same concept as employed by the U.S. Army's Blue Force Tracker 
system, for example.) 
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References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
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have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 11 on that same page). In Tribes, 
as shown in the screenshots above, each team has different color arrows to point to 
teammates and enemies, which convey information on player location, when they are in 
the player's field of view. 

Further, the 3-D arrow disclosed in Thomas is of a significantly different size and 
shape than that used by Tribes to indicate player position. This would meet the recited 
limitation of "having an attribute different than that of other annotation." Also, as noted 
in other rejections, Piekarski includes a forearm keyboard that would prima facie allow 
the user to input annotation that would update the system with regards to the nature of 
the attention target. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski and Miyashita with the arrows of Thomas and Tribes, since 
Piekarski discloses tracking arrows to indicate objects (e.g. the helicopter in Figs. 8B 
and 8C) and target annotation in Fig. 7 and Thomas would allow other users / soldiers / 
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players (in the case of T2000) to see the targets of interest and guide their attention to 
them while indicating known friendly and unfriendly targets (Tribes) when the targets 
were inside their field of view. 

29. As to claim 26, 

An information processing method according to claim 22, wherein an annotation 
indicating a position of other user is generated and merged to the synthesized image. 

This claim is a substantial duplicate of claim 24; therefore, the rejection for claim 
20 is incorporated herein by reference. The main difference is that "Tribes!" shows on 
the screenshots above that there are arrows pointing to friendly and unfriendly targets in 
the user's field of view (that is, other users are the targets). It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to combine the 
combine the augmented reality of T2000 and the annotation of Piekarski with the arrows 
of Thomas and Tribes, since Piekarski discloses tracking arrows to indicate objects (e.g. 
the helicopter in Figs. 8B and 8C) and target annotation in Fig. 7 and Thomas would 
allow other users / soldiers / players (in the case of T2000) to see the targets of interest 
and guide their attention to them while indicating known friendly and unfriendly targets 
(Tribes) when the targets were inside their field of view. 

Allowable Subject Matter 

30. Claims 19 and 25 are allowed for the reasons set forth in the previous office 
action, as applicant amended said claim to incorporate all the limitations of the parent 
independent and intervening claims and made the claim independent. 
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Conclusion 

31 . Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Eric V Woods whose telephone number is 571-272- 
7775. The examiner can normally be reached on M-F 7:30-4:30 alternate Fridays off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi can be reached on 571-272-7664. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 





Eric Woods 
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